Method and system for handling streaming information

ABSTRACT

One aspect of the present invention leads to a method of handling streaming information. The method includes receiving the streaming information and analyzing the streaming information to locate one or more points of interest in the streaming information. An index of the one or more points of interest are generated. The index is delivered to a user separate from the streaming information. Rendering of the streaming information is controlled based on the points of interest.

CROSS REFERENCE TO PENDING APPLICATION

The present application is a continuation of and claims priority of U.S.patent application Ser. No. 10/827,619, filed on Apr. 19, 2004, which isa continuation of U.S. patent application Ser. No. 09/286,808, filedApr. 6, 1999, now U.S. Pat. No. 6,748,481, issued Jun. 8, 2004; thecontents of both aforementioned applications are hereby incorporated byreference in their entirety.

BACKGROUND OF THE INVENTION

The present invention relates to streaming information. Moreparticularly, the present invention relates to recording streaminginformation and retrieving the stored information for selectiveplayback.

With advances in technology including faster processors, improvedgraphics, and so forth, a desktop computer can easily receive andpresent streaming information to the user. Common examples of streaminginformation include streaming video and audio delivered over a wide areanetwork, such as the Internet. For instance, television broadcastsignals that would otherwise be transmitted wirelessly using satellites,television transmitters, etc., are encoded and made available fortransmission to remote computer users via the Internet. Upon request bythe desktop computer user, the encoded data packets containing audio andvideo data are sent to the desktop computer user sequentially. Uponreceipt, the data packets are decoded and processed by the desktopcomputer in order to render the streaming information to the user in asclose to real time as possible. After rendering or presentation, thedata packets are discarded.

Although processing streaming information in the manner described aboveis useful, there exist a number of shortcomings. Currently, streaminginformation is provided at the request of each desktop computer. Thus,each user must form a separate connection with the source of streaminginformation in order to receive the desired streaming information. Onceinitiated, the user is unable to control the manner in which streaminginformation is rendered. For instance, the user cannot temporarily“pause” the incoming streaming information in order to perform anothertask and then resume viewing when desired. Likewise, the user is unableto repeat a previously rendered portion since the data packets arediscarded, or skip ahead since the data packets have not been received.

There thus is an ongoing need to improve the manner in which streaminginformation is rendered. Although described above with respect to adesktop computer and streaming information received from the Internet,the improved process should be applicable to other informationappliances or computing devices and other forms of streaminginformation.

SUMMARY OF THE INVENTION

One aspect of the present invention relates to a method of handlingstreaming information. The method includes receiving the streaminginformation and analyzing the streaming information to locate one ormore points of interest in the streaming information. An index of theone or more points of interest are generated. The index is delivered toa user separate from the streaming information. Rendering of thestreaming information is controlled based on the points of interest.

Another aspect of the present invention relates to a method for handlingstreaming information. Streaming information is received that is relatedto a segment of multiple streams of streaming information. A portion ofthe segment is rendered and the entire segment is archived to a morepermanent file after the portion of the segment has been rendered.

Another aspect of the present invention relates to a computer readablemedium encoded with a data structure having a data block format forstreaming information. The data block format includes a data blockhaving a header portion and an end portion. A first data block sizefield is included in a header portion indicating the size of the datablock and a second data block size field is included in the end portionindicating the size of the data block. A payload field is bounded by thefirst data block size field and the second data block size field. Thepayload field includes streaming information to be rendered.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a first exemplary environment for thepresent invention.

FIG. 2 is a block diagram-of a second exemplary environment for thepresent invention.

FIG. 3 is a perspective view of a mobile device.

FIG. 4 is a block diagram of a third exemplary environment for thepresent invention.

FIG. 5 is a system for processing streaming information.

FIG. 6 is a block diagram of a delay filter.

FIG. 7 is a block diagram of delay filter 112 showing an interface to acircular buffer in the filter.

FIG. 8 is a diagram illustrating the circular buffer 124 along a lineartime line.

FIGS. 9A and 9B together form a flow chart for a write portion of asynchronization algorithm according to one embodiment of the presentinvention.

FIG. 10 is a diagram illustrating advancement of a Tail Pointer variablewithin the circular buffer.

FIGS. 11A and 11B together illustrate a flow chart for a read portion ofthe synchronization algorithm according to one embodiment of the presentinvention.

FIG. 12 is a diagram illustrating an example of the circular buffer whena reader module has overtaken a writer module.

FIG. 13 is a diagram illustrating a status register which is maintainedfor each data channel for specifying which user operations are permittedor prohibited in a present state of the channel.

FIG. 14 is a table listing examples of user operations corresponding touser operation fields UOP0 -UOP10 in FIG. 13.

FIG. 15 is a flow chart of a software object that is called by anapplication in response to a user operation request.

FIG. 16 is a flow chart illustrating steps performed by the delay filterin maintaining the user operation bits of the status register shown inFIG. 13.

FIG. 17 is a pictorial representation of a data block.

FIG. 18 is a pictorial representation of a first sequence of datablocks.

FIG. 19 is a pictorial representation of a second sequence of datablocks.

FIG. 20 is a block diagram of another embodiment of a system inaccordance with the present invention.

FIG. 20A is a block diagram of an example of a filter graph.

FIGS. 21A-21D are block diagrams illustrating stream analysis.

FIGS. 22-23C are flow diagrams illustrating stream analysis.

FIG. 24 is a flow diagram illustrating indexing.

FIG. 25 is an illustration of a portion of a data buffer.

FIG. 26 is a flow diagram illustrating a seeking operation.

DETAILED DESCRIPTION OF THE ILLUSTRATIVE EMBODIMENTS Overview

FIG. 1 and the related discussion are intended to provide a brief,general description of a first exemplary computing environment in whichthe invention may be implemented. Although not required, the inventionwill be described, at least in part, in the general context of processorexecutable instructions, such as program modules being executed by acontroller, processor, a personal computer or other computing device.Generally, program modules include routine programs, objects,components, data structures, etc., that perform particular tasks orimplement particular abstract data types. Tasks performed by the programmodules are described below and with the aid of block diagrams andflowcharts. Those skilled in the art can implement the description,block diagrams and flowcharts to processor executable instructions,which can be written on computer readable mediums. In addition, thoseskilled in the art will appreciate that the invention may be practicedwith other information appliances, including hand-held devices,multiprocessor systems, microprocessor-based or programmable consumerelectronics, network PCs, minicomputers, mainframe computers, and thelike. The invention is also applicable in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed computingenvironment, program modules may be located in both local and remotememory storage devices.

With reference to FIG. 1, the first exemplary environment for theinvention includes a general purpose computing device in the form of aconventional personal computer 20, including processing unit 21, asystem memory 22, and a system bus 23 that couples various systemcomponents including the system memory to processing unit 21. System bus23 may be any of several types of bus structures including a memory busor memory controller, a peripheral bus, and a local bus using any of avariety of bus architectures. The system memory includes read onlymemory (ROM) 24 and random access memory (RAM) 25. A basic input/outputsystem 26 (BIOS), containing the basic routine that helps to transferinformation between elements within personal computer 20, such as duringstart-up, is stored in ROM 24. Personal computer 20 further includes ahard disk drive 27 for reading from and writing to a hard disk (notshown), a magnetic disk drive 28 for reading from or writing toremovable magnetic disk 29, and an optical disk drive 30 for readingfrom or writing to a removable optical disk 31 such as a CD ROM or otheroptical medium. Hard disk drive 27, magnetic disk drive 28, and opticaldisk drive 30 are connected to system bus 23 by a hard disk driveinterface 32, magnetic disk drive interface 33, and an optical driveinterface 34, respectively. The drives and the associated computerreadable medium provide nonvolatile storage of computer readableinstructions, data structures, program modules and other data forpersonal computer 20.

Although the exemplary environment described herein employs a hard disk,a removable magnetic disk 29 and a removable optical disk 31, it shouldbe appreciated by those skilled in the art that other types of computerreadable medium which can store data that is accessible by a computer,such as magnetic cassettes, flash memory cards, digital video disks,Bernoulli cartridges, random access memories (RAMs), read only memory(ROM), and the like, may also be used in the exemplary operatingenvironment.

A number of program modules may be stored on hard disk, magnetic disk29, optical disk 31, ROM 24 or RAM 25, including an operating system 35,one or more application programs 36, other program modules 37, andprogram data 38. A user may enter commands and information into personalcomputer 20 through input devices such as a keyboard 40 and pointingdevice (mouse) 42. Other input devices (not shown) may include amicrophone, joystick, game pad, satellite dish, scanner, or the like.These and other input devices are often connected to processing unit 21through a serial port interface 46 that is coupled to system bus 23, butmay be connected by other interfaces, such as a sound card, a parallelport, a game port or a universal serial bus (USB). A monitor 47 or othertype of display device is also connected to system bus 23 via aninterface, such as a video adapter 48. In addition to monitor 47,personal computers may typically include other peripheral output devicessuch as a speaker 49 connected to a sound card 57 and printers (notshown).

Personal computer 20 may operate in a networked environment using logicconnections to one or more remote computers, such as a remote computer49. Remote computer 49 may be another personal computer, a server, arouter, a network PC, a peer device or other network node, and typicallyincludes many or all of the elements described above relative topersonal computer 20, although only a memory storage device 50 has beenillustrated in FIG. 1. The logic connections depicted in FIG. 1 includea local are network (LAN) 51 and a wide area network (WAN) 52. Suchnetworking environments are commonplace in offices, enterprise-widecomputer network Intranets and the Internet.

When used in a LAN networking environment, personal computer 20 isconnected to local area network 51 through a network interface oradapter 53. When used in a WAN networking environment, personal computer20 typically includes a modem 54 or other means for establishingcommunications over wide area network 52, such as the Internet. Modem54, which may be internal or external, is connected to system bus 23 viaserial port interface 46. In a network environment, program modulesdepicted relative to personal computer 20, or portions thereof, may bestored in the remote memory storage devices. It will be appreciated thatthe network connections shown are exemplary and other means ofestablishing a communications link between the computers may be used.

Streaming information can be received by computer 20 using a number ofknown methods and technologies. Commonly, a source of streaminginformation is a remote computer wherein computer 21 is connected to theremote computer using a wired or wireless modem. This technique is usedoften when streaming information is provided through an intranet or theInternet, typically in digital form. Digital streaming information canfurther comprise satellite signals received by a satellite receiver,dish or the like.

However, streaming information can also arrive as analog signals. Forinstance, the streaming information can also comprise broadcast radio ortelevision signals. In such cases, computer 20 includes a radio tuner 60and a television tuner 62 to receive the broadcast signals and convertthe analog signals to digital form for transmission over system bus 23.

It should be understood that the present invention can be used in othercomputing devices besides computer 20, discussed above. FIG. 2 is ablock diagram of a mobile device 68, which is another exemplarycomputing environment. Mobile device 68 includes a microprocessor 70,memory 72, input/output (I/O) components 74, a communication interface76 for communicating with, for example, computer 20. In a oneembodiment, the afore-mentioned components are coupled for communicationwith one another over a suitable bus 78.

Memory 72 is implemented as non-volatile electronic memory such asrandom access memory (RAM) with a battery back-up module (not shown)such that information stored in memory 72 is not lost when the generalpower to mobile device 68 is shut down. A portion of memory 72 ispreferably allocated as addressable memory for program execution, whileanother portion of memory 72 is preferably used for storage, such as tosimulate storage on a disc drive.

Memory 72 includes an operating system 80, an application program 82 aswell as an object store 84. During operation, operating system 80 ispreferably executed by processor 70 from memory 72. Operating system 80,in one preferred embodiment, is a “WINDOWS CE” brand operating systemcommercially available from Microsoft Corporation. Operating system 80is preferably designed for mobile devices, and implements databasefeatures which can be utilized by application 82 through a set ofexposed application programming interfaces and methods. The objects inobject store 84 are maintained by application 82 and operating system80, at least partially in response to calls to the exposed applicationprogramming interfaces and methods.

Communication interface 76 represents numerous devices and technologiesthat allow mobile device 68 to receive streaming information. Thedevices are similar to those discussed above with respect to computer 20and include wired and wireless modems, satellite receivers and broadcasttuners to name a few. Mobile device 68 can also be directly connected tocomputer 20 to exchange data therewith. In such cases, communicationinterface 76 can be an infrared transceiver or a serial or parallelcommunication connection, all of which are capable of transmittingstreaming information.

FIG. 3 is a simplified pictorial illustration of mobile device 68.Mobile device 68 can be a desktop assistant sold under the designationH/PC having software provided by the Microsoft Corporation. In oneembodiment, mobile device 18 includes a miniaturized keyboard 83,display 85 and stylus 86. In the embodiment shown in FIG. 3, display 85is a liquid crystal display (LCD) which uses a contact sensitive displayscreen in conjunction with stylus 86. Stylus 86 is used to press orcontact display 85 at designated coordinates to accomplish certain userinput functions. Miniaturized keyboard 83 is preferably implemented as aminiaturized alpha-numeric keyboard, with any suitable and desiredfunction keys, which are provided for accomplishing certain user inputfunctions. In other known embodiments, keyboard 83 is omitted and a“soft” keyboard is provided through the contact sensitive displayscreen. In yet other embodiments, a character recognition module isemployed to recognize characters written on the contact sensitivedisplay screen using stylus 86.

FIG. 4 illustrates yet another exemplary environment in which thepresent invention can operate. In FIG. 4, an entertainment appliance 90is illustrated and includes processing unit 21, system memory 22 andsystem bus 23. Hard disk drive 27, or any of the other storage devicesdescribed above, are further coupled to system bus 23 and used fortemporary and permanent storage of program applications, data, etc.Unlike typical desktop computers such as computer 20 described above,entertainment appliance 90 may use a limited input device such as ahandheld remote 92 operable with a receiver 94, which can be an infraredreceiver, wireless receiver, or the like. In entertainment appliance 90,information is rendered to the user using monitor 47 or other displaydevice that is coupled to system bus 23 with video adapter 48. Audioinformation is also rendered and illustrated herein with speaker 49.Speaker 49 is coupled to system bus 23 with sound card 57, which can becombined with video adapter 48 to form a signal device, if desired. Itshould be further understood that audio and video information could beprovided to external components, such as amplifiers or the like, whichin turn, are connected to monitor 47 and speakers 49.

Streaming information is provided to entertainment appliance 90 througha communications interface 98. Communications interface 98 can be any ofthe devices and technologies described above with respect to the twoprevious environments.

Delay Filter

FIG. 5 is a general block diagram illustrating a system 110 forprocessing streaming information. System 110 includes a delay filter 112for temporarily storing streaming information received from a streaminginformation source 114. Delay filter 112 is further coupled to arendering device or devices 116 to render streaming information uponrequest by the user. Also illustrated in FIG. 5 is an encoder 118 and adecoder 120. Although not required, encoder 118 and decoder 120 canimprove system performance wherein encoder 118 receives streaminginformation source 114 and compresses the streaming information prior totransfer and storage in delay filter 112. Decoder 120 receives thestreaming information temporarily stored in delay filter 112 in thecompressed format and uncompresses the streaming information prior totransferring the streaming information to rendering device 116.

At this point, it should be noted that system 110 can be operated in anyof the computing environments described above, or similar computingenvironments. Those skilled in the art will appreciate that delay filter112, rendering device 116, encoder 118 and decoder 120 can beimplemented in hardware, software, or combinations thereof. In oneembodiment by way of example, delay filter 112 is embodied in theoperating system. Higher level application programs or other portions ofthe operating systems can access functions of delay filter 112 usingapplication program interfaces (APIs) as is well known in the art.

In operation, streaming information source 114 provides an informationstream to delay filter 112 (optionally through encoder 118). Generally,the streaming information comprises digital data representing one ormore channels of content information. For instance, streaminginformation source 114 can comprise an Intranet or the Internetavailable through the communication interfaces described above.Likewise, streaming information source 114 can comprise an analog ordigital television tuner wherein separate audio, video and data (e.g.closed captioning) information streams comprise a single channel. Othersources of streaming information include, but are not limited to, audiotuners, satellite receivers and the like.

In the embodiment illustrated, encoder 118 receives the streaminginformation and encodes or compresses the streaming information into aknown format such as “MPEG”, “AVI”, “MOV” (Apple®QuickTime®) and “WAV”,although, if used, the present invention is not limited to any oneparticular encoding format.

Generally, as discussed below, delay filter 112 includes a writer module122, a circular buffer 124 and one or more reader modules 126. Writermodule 122 receives the streaming information provided by streaminginformation source 114 and writes the streaming information intocircular buffer 124. Circular buffer 124 can comprise any of the storagedevices described above, for example, hard disk 27 or RAM memory. Readermodule 126 accesses circular buffer 124 to retrieve the streaminginformation when the streaming information is to be rendered. If thestreaming information stored in circular buffer 124 is encoded orcompressed, decoder 120 decodes or uncompresses the streaminginformation, which is then provided to rendering device 116.

FIG. 6 is a more detailed pictorial representation of delay filter 112.In this illustrated embodiment, the streaming information comprises atelevision signal or channel that includes audio, video and data (closedcaptioning) streams. The streaming information is first provided to astream analyzer 130, which analyzes the incoming streams and providessuch information as synchronization points that may be present in eachof the streams. Synchronization points are used in rendering some typesof streaming information and are discussed in detail below.

Streaming information and synchronization point information is providedto writer module 122. In one embodiment, as illustrated, writer module122 comprises a mux writer that receives multi-stream streaminginformation for storage in circular buffer 124. As indicated above, oneor more reader modules 126 (herein labeled as 1261, 1262 and 1263) areprovided to read the streaming information from circular buffer 124 forrendering. Writer module 122 stores synchronization information in anindex 132. Reader modules 126 may access index 132 in order to locate aparticular portion of the streaming information and properly render thestreaming information. Operation of writer module 122, circular buffer124, reader modules 126 and index 132 are discussed in detail below.

In the embodiment illustrated, two separate playback reader modules 126₁, and 126 ₂ are illustrated having outputs that provide streaminginformation to separate video, audio and data decoders 120 and renderingdevices 116. In general, this illustrates that separate reader modules126 can be reading streaming information from circular buffer 124 atdifferent points in circular buffer 124, and thus, represents separateindividuals accessing the data stored therein. In addition, other readermodules 126, such as indicated at 136, can be implemented to archive andstore the streaming information in circular buffer 124 for laterviewing. Generally, in archiving system 136, reader module 126 ₃provides streaming information to a mux formatter 138 that, in turn,provides the information to a writer module 140 for storage in any ofthe storage devices indicated above such as hard disk 27.

Circular Buffer

Referring back to circular buffer 124, circular buffer 124 has“floating” beginning and ending points, which are referred to as alogical “head” 150 and “tail” 152. Head 150 corresponds to the logicalhead of valid data in circular buffer 124, and tail 152 corresponds tothe logical tail of valid data in circular buffer 124. Writer module 122always writes to the head of buffer 124, which moves circularly throughthe buffer in the direction of arrow 154. Buffer 124 therefore alwayshas a fixed maximum time quantum of data available for reading. Forexample, when time shifting multimedia (e.g., audio and video) content,writer module 122 receives the streaming multimedia information andstores the information in circular buffer 124. The user views the storedmultimedia content through one of the readers modules 126. The circularstructure of buffer 124 allows some portion of the streaming informationto be available for the user to “instant replay” or “pause” on demand,for example, without allowing the buffer to “fill-up” with“time-shifted” data. Circular buffer 124 can be implemented in volatileor non-volatile memory, such as random access memory (RAM), a hard disk,a floppy disk or an optical disk. In one embodiment, circular buffer 124is implemented in hard disk drive 27.

FIG. 7 is a block diagram of delay filter 112 showing an interface tocircular buffer 124. Buffer IO layer 200 interfaces between circularbuffer 124 and the clients of the buffer, which include writer module122 and reader modules 126 (labeled 126 ₁-126 _(N), where N is aninteger variable greater than or equal to 1). Buffer IO layer 200implements the circularity of buffer 124 and synchronizes writer module122 with reader modules 126 ₁-126 _(N) Buffer IO layer 200 implementscircularity by translating between logical addresses, used at upperinterfaces 204-207 to identify logical positions within buffer 124, andphysical (wrap around) addresses, used at lower interface 208 foridentifying particular physical addresses within buffer 124. The logicaladdresses always increase (or decrease) with time and never wrap around.The logical addresses can be periodically reset as desired. The physicaladdresses wrap around (i.e. wrap from the highest address of the bufferto the lowest address of the buffer) at a frequency determined by thecircular buffer size. As each successive block of the streaminginformation is received by writer module 122, the block is associatedwith a respective logical address or a range of logical addresses, whichincreases with each successive block. Buffer IO layer 200 translates thelogical addresses into corresponding physical addresses which are usedto access circular buffer 124. In one embodiment, buffer IO layer 200translates the logical addresses into physical addresses as a functionof the logical address modulo the buffer size (i.e. the number ofstorage locations in circular buffer 124). Buffer layer IO 200 can beimplemented in an operating system, for example.

Write/Read Synchronization

Since writer module 122 and reader modules 126 ₁-126 _(N) can operateindependently of one another and at different data rates, buffer IOlayer synchronizes writer module 122 and reader modules 126 ₁-126 _(N)to maintain a predetermined temporal order between writing and reading.In one embodiment, buffer IO layer 200 prevents any reader 126 ₁-126_(N) from reading data that is not yet logically available and preventswriter module 122 from overwriting data that is in the process of beingread by one or more of the reader modules 126 ₁-126 _(N). In circularbuffer 124, a given physical position corresponds to multiple logicalpositions. Without synchronization, a reader module that is trailingwriter module 122 by a distance that is close to the buffer size may bereading from the same physical area to which writer module 122 iswriting. Buffer IO layer 200 also allows reader modules 126 ₁-126 _(N)to follow writer module 122 as close as possible to minimize latency.

Buffer IO layer 200 implements a synchronization algorithm for writermodule 122 and reader modules 126 ₁-126 _(N). Each time writer module122 wants to pass data to buffer IO layer 200, its correspondingapplication calls the synchronization algorithm. Similarly, each timeone of the reader modules 126 ₁-126 _(N) wants to read data from bufferIO layer 200, its corresponding application calls the synchronizationalgorithm. The synchronization algorithm can be implemented in hardware,software, or a combination of both, as desired.

The synchronization algorithm uses “blocking” to (1) block a readermodule 126 ₁-126 _(N) that is trying to read data which has not yet beenwritten and/or (2) block writer module 122 if it is trying to write toan area of circular buffer 124 from which one of the reader modules 126₁-126 _(N) is currently reading. In both cases, one component is blockeduntil another component has completed the operation necessary to removethe offending condition. For example, if writer module 122 is blocked,it remains blocked until all of the reader modules 126 ₁-126 _(N), whichwho are reading from the area to be written, have completed their reads.When a reader module is blocked, it remains blocked until writer module122 has written all of the data requested by that reader module.

The synchronization algorithm uses a plurality of shared variables. Eachclient has its own set of variables which are shared with the otherclients. In FIG. 8, circular buffer 124 is arranged along a linear timeline from zero to infinity. The synchronization algorithm maintains a“Tail Pointer” 230 which is an integer variable indicating the logicaltail of valid data in circular buffer 124. A “Head Pointer” 232 is aninteger variable which indicates the logical head of valid data incircular buffer 124. For a read operation to succeed, the logical readposition within circular buffer 124 must be greater than or equal toTail Pointer 230 and less than or equal to Head Pointer 232. Sincebuffer 124 is circular, the logical position of Tail pointer 230 is alsologically “ahead” of the logical position of Head Pointer 232.

When writer module 122 issues a write command, it specifies an area inbuffer 124 to be written, starting at a present write position 234. A“Writer Blocked On” integer variable 236 is used to identify the logicalposition corresponding to the end of the data to be written when all orpart of the area to be written is blocked by one or more of the readermodules 126 ₁-126 _(N). The Writer Blocked On variable is set when oneof the reader modules 126 ₁-126 _(N) is currently reading from thatarea. For example, a reader module may be presently reading from logicalposition 235, which is in the area between positions 234 and 236. A“zero” value for the “Writer Blocked On” variable indicates that writermodule 122 is not currently blocked by any of the reader modules 126₁-126 _(N).

A “Currently Reading From” integer variable is maintained for eachreader module 126 ₁-126 _(N). The Currently Reading From variable isused to indicate that the reader module is currently performing a readthat starts at this logical position in circular buffer 124. Forexample, in FIG. 8, a reader module is currently reading from logicalposition 235 in circular buffer 124. The Currently Reading From variableis used to prevent writer module 122 from overwriting the data inlogical position 235 while the reader module is reading from logicalposition 235. When a particular reader module 126 ₁-126 _(N) is notcurrently reading from circular buffer 124, its corresponding CurrentlyReading From variable is set to infinity.

A “Writer Unblocked Event” variable is used to “wake-up” writer module122 when it can proceed with its desired write command. For example, theWriter Unblock Event variable is set to an active state when writermodule 122 is not blocked, and is reset to an inactive state when writermodule 122 is blocked. The Writer Unblock Event can be implemented as aWindows® event (a “Win32” event) or any other similar synchronizationmechanism familiar to those skilled in the art.

A “Reader Unblock Event” variable is used for each reader module 126₁-126 _(N) for “waking-up” the reader module when the data it isrequesting is available. For example, the Reader Unblock Event variableis set to an active state when the corresponding reader module 126 ₁-126_(N) is not blocked and is reset to an inactive state when thecorresponding reader module 126 ₁-126 _(N) is blocked.

A “Critical Section” variable is used to protect access to each of theabove shared variables. For example, a “Win32” Critical Section may beused or any other similar synchronization mechanism such as a mutualexclusion “mutex” object, as is known to those skilled in the art.

FIGS. 9A, 9B, 11A and 11B together form a flow chart for thesynchronization algorithm according to one example of the presentinvention. The steps performed during a write are shown in FIGS. 9A andare labeled 300-317. At step 300, when the application driving writermodule 122 wishes to pass data to Buffer IO layer 200, the applicationcalls algorithm 290. At step 301, algorithm 290 locks the CriticalSection to protect access to the variables used in steps 302-305. Atstep 302, algorithm 290 advances the “Tail Pointer” to the logicalposition in circular buffer 124 that corresponds to the end of the writecommand. This logical position will depend upon the amount of data beingwritten by writer module 122. The length of data can vary from one writecommand or data block to the next, and the data can have arbitrary dataformats, which can also vary from one write command or data block to thenext.

FIG. 10 is a diagram illustrating advancement of the Tail Pointer incircular buffer 124. Tail Pointer 250 is advanced from logical position260 to logical position 262. Advancing Tail Pointer 250 immediatelyinvalidates the area to be written (behind the advanced Tail Pointer250) for future reads, even if writer module 122 must wait before it canactually start writing the data. As a result, once writer module 122notifies buffer IO layer 200 that it wants to write to an area incircular buffer 124 by calling the writer algorithm, no new readermodules can start reading from that area. This minimizes the time duringwhich writer module 122 must wait for reader modules 126 ₁-126 _(N) andprevents the writer from being perpetually blocked.

Referring back to FIG. 9A, synchronization algorithm 290 determineswhether any of the “Currently Reading From” variables of readers 126₁-126 _(N) is less than (i.e. behind) the “Tail Pointer” variable, atstep 303. For example, in FIG. 10, a reader module may have a CurrentlyReading From variable 270 which is pointing to a logical position 272 incircular buffer 124 that is less than the logical position 262 of theadvanced Tail Pointer 250. If this is the case, synchronizationalgorithm 290 blocks or delays writer module 122 so that the conflictingreader module can “get out of the way”. If none of the reader modules'“Currently reading From variable is less than the advanced Tail Pointervariable, then synchronization algorithm 290 proceeds directly to step311 (shown in FIG. 9B) to write the data to circular buffer 124.

At step 304, algorithm 290 sets the “Writer Blocked On” variable to thevalue of the “Tail Pointer” variable 250. This indicates the largestlogical position in circular buffer 124 at which the reader modules 126₁-126 _(N) must be clear of before writer module 122 can write the data.Algorithm 290 then blocks writer module 122 by resetting the “WriterUnblock” event to the inactive state, at step 305. Since writer module122 is blocked, the Critical Sections are unlocked at step 306. At step307, algorithm 290 waits for the “Writer Unblock” event variable to beactivated by the synchronization algorithm for the conflicting readerthat is accessing the area to be written. When the conflicting readermodule activates the “Writer Unblock” event, algorithm 290 locks theCritical Section at step 308 and then resets the “Writer Blocked On”variable to zero, at step 309. Algorithm 290 again unlocks the CriticalSection at step 310 and proceeds to step 311 (shown in FIG. 9B).

At step 311, buffer IO layer 200 converts the starting logical addressto a circular or physical address and writes the data in circular buffer124, beginning at that physical address. Once the data has been written,algorithm 290 locks the Critical Section for the variables used in steps313-315 and advances the “Head Pointer” variable to the logical positionin circular buffer 124 that corresponds to the end of the data writtenby writer module 122. In the example shown in FIG. 10, Head Pointer 274is advanced from position 276 to position 278 (which can also be viewedas being “behind” the advanced Tail Pointer 250). Advancing Head Pointer274 validates the newly written data between Head Pointer 274 and TailPointer 250.

At step 314, algorithm 290 determines whether any of the “Reader BlockedOn” variables for reader modules 126 ₁-126 _(N) is less than theadvanced “Head Pointer” variable 274. If not, none of the reader modules126 ₁-126 _(N) were waiting for the data written by writer module 122,and algorithm 290 proceeds to step 316. If so, one or more of the readermodules 126 ₁-126 _(N) were waiting for writer module 122, which has nowvalidated the desired logical positions. Algorithm 290 sets any such“Reader Unblock” event variables at step 315 to unblock thecorresponding reader module.

In the example shown in FIG. 10, a reader module may be waiting to readdata at logical position 280 which was not yet available when HeadPointer 274 was pointing to logical position 276. That reader modulewould have had a “Reader Blocked On” variable pointing to logicalposition 280, as shown by arrow 282. Now that “Head Pointer” variable274 is pointing to logical position 278, which is ahead of logicalposition 280, the data in position 280 is available for reading, andalgorithm 290 sets the “Reader Unblock” event variable for that readermodule to the active state, thereby unblocking the reader module.

Referring back to FIG. 9B, the corresponding Critical Sections areunlocked at step 316, and the algorithm completes at step 317.

FIGS. 11A and 11B together illustrate a read portion of synchronizationalgorithm 290, which includes steps 351-372. When one of the readermodules 126 ₁-126 _(N) desires to read data from circular buffer 124,that reader module calls algorithm 290 in buffer IO layer 200, at step351. At step 352, algorithm 290 locks the Critical Section to protectthe variables used in steps 353-355. At step 353, algorithm 290determines whether the amount of data to be read extends to a logicalposition in circular buffer 124 which is beyond the logical position ofthe “Head Pointer” such that a portion of the desired data is not yetvalid. If not, algorithm 290 proceeds directly to step 360.

FIG. 12 is a diagram illustrating an example of circular buffer 124 whenthe requested data extends beyond the Head Pointer. Circular buffer 124has a Tail Pointer 330 at a logical position 332 and a Head Pointer 334at a logical position 336. If the reader module requests data beginningat logical position 337 and extending up to logical position 338, whichis beyond the logical position 336 of Head Pointer 334, then algorithm350 sets the “Reader Is Blocked On” variable for that reader module tothe logical position corresponding to the end of the requested read data(e.g. logical position 338), as shown by arrow 340. Referring back toFIG. 11A, algorithm 350 resets the “Reader Unblock” event variable toblock the corresponding reader module at step 355. Since the readermodule is blocked, algorithm 350 unlocks the corresponding CriticalSection, at step 356. At step 357, algorithm 350 waits for the “ReaderUnblock” variable to be set for this reader (at step 315 in FIG. 9B).Once the “Reader Unblock” variable is set, the corresponding readermodule is unblocked and the Critical Section is locked at step 358. The“Reader Blocked On” variable for that reader is then reset to infinity,at step 359, and algorithm 350 proceeds to step 360.

At step 360, algorithm 350 determines whether the logical positioncorresponding to the beginning of the requested data is before thelogical position of the “Tail Pointer” variable. If so, the requesteddata is invalid since it has already been overwritten by writer module122. In this case, algorithm 350 unlocks the Critical Section at step361 and fails the requested read operation at step 362. The failure canbe passed to the corresponding reader module through a variety ofmechanisms, such as a status variable.

If the requested data starts at a logical position that is not beforethe “Tail Pointer” variable, algorithm 350 proceeds to step 363 (shownin FIG. 11B). At step 363, algorithm 350 sets the “Currently ReadingFrom” variable of the reader to the logical position at the start of therequested data. The Critical Section is unlocked at step 364, and thestarting logical address is converted to a circular or physical addressfor circular buffer 124 (shown in FIG. 7). At step 365, buffer IO layer200 reads the requested amount of data, beginning at the convertedlogical address and provides the data to the corresponding readermodule, at step 365. At step 366, algorithm 350 locks the CriticalSection. At step 367, the “Currently Reading From” variable for thereader is reset to infinity since the read operation has completed.

At step 368, algorithm 350 determines whether the “Writer Blocked On”variable points to a logical position in circular buffer 124 that isgreater than the read position. The read position is the logicalposition in circular buffer 124 corresponding to the beginning of thedata to be read. If the “Writer Blocked On” variable is not greater thanthis logical position, then this reader module was not blocking writingmodule 122. Algorithm 350 then unlocks the Critical Section at step 369and completes its function at step 370.

If the “Writer Blocked On” variable is greater than this position, thenthis reader module was blocking writing module 122, and algorithm 350proceeds to step 371 to determine whether any other reader module isblocking writer module 122. If not, algorithm 350 sets the “WriterUnblock” event variable to unblock writer module 122, at step 372. Ifthere are other reader modules that are blocking writer module 122, thenalgorithm 350 proceeds to step 369 without setting the “Writer Unblock”event variable.

The algorithm shown in FIGS. 9 and 11 can be modified in a variety ofways in alternative embodiments. For example, it may not be desirable toblock writer module 122. If the streaming information being provided towriter module 122 may be lost if writer module 122 is delayed by anysignificant period of time, such as when writer module 122 is coupled toa television tuner, then it would not be desirable to delay writermodule 122. In these embodiments, algorithm 290 is modified in FIGS. 9Aand 9B to remove steps 303-310 which serve to delay writer module 122.Algorithm 290 proceeds from step 302 directly to step 311. Similarly,algorithm 290 is modified in FIGS. 11A and 11B to remove steps 366-369,371 and 372 which serve to selectively unblock writer module 122.

In addition, algorithm 290 can be modified to perform steps 360-362(validating the read operation) either before or after the read isperformed at step 365, or both. Validating the read operation before theactual read operation avoids a wasteful read. However, steps 360-362 canbe performed after step 366 if desired. Steps 360-362 can also beperformed both before and after read step 365.

The synchronization algorithm shown in FIGS. 9 and 11 does not require acircular buffer such as that shown in FIG. 6. The synchronizationalgorithm can also be used with minor modification in a linear bufferhaving both logical and physical addresses that wrap around at afrequency based on the size of the buffer. However, a circular bufferallows a reader module to be time-shifted with respect to the writermodule such that a portion of the streaming information is available tothe user for “instant replay” on demand. A circular buffer allows theuser to pause a program, but does not allow the user to inadvertentlyfill up the buffer with time shifted data. The writer module is alwayswriting to the “beginning” of the buffer, which is point that movescircularly through the buffer.

The synchronization algorithm shown in FIGS. 9 and 11 also allows theuser to fast forward through the data stored in the buffer. If the userhas paused a reader module so that the writer module is considerablyahead of that reader, the user or the application has the option offast-forwarding the reader module to catch up to the writer module.However, the synchronization algorithm stops fast-forwarding when thereader module catches up with the writer module. For example, the readermodule may initiate a “DirectShow” event (or other appropriate eventmechanism available to the software environment) whenever the readermodule is blocked by the writer module. The application detects thisevent, and switches the state of the reader module from fast-forward tonormal play mode. If the writer module catches up to the reader module,corruption by the writer module can be avoided by (1) blocking thereader until the data is available, (2) blocking the writer until thereader is out of the way, or (3) moving the reader ahead by unpausing orfast forwarding the reader.

User Operation Permission Checking

In one embodiment, delay filter 112 (shown in FIG. 6) further includes auser operation permission checking and reporting mechanism whichmaintains consistent operating states in the filter. FIG. 13 is adiagram illustrating a status register 420 which is maintained by delayfilter 112 for each data stream, or channel, through the filter. Delayfilter 112 can maintain register 420 in RAM, for example, or any otherstorage medium. Status register 420 includes a plurality of fields, witheach field having one or more bits. In the example shown in FIG. 13,bits 0-10 of status register 420 correspond to user operation fieldsUOP1-UOP10, respectively. Bits 11-27 correspond to reserved fields. Bits28-31 correspond to a channel streamer ID field which identifies theparticular data stream through writer module 122 and reader modules 126₁-126 _(N) to which status register 420 is associated. Each useroperation field UOP0-UOP10 has a binary value which indicates whetherthe corresponding user operation is permitted or prohibited. Forexample, a binary “one” in the corresponding field would indicate thatthat user operation is permitted. A binary “zero” in a user operationfield would indicate that that user operation is prohibited. FIG. 14 isa table listing examples of user operations that correspond to useroperation fields UOP0-UOP10. Delay filter 112 maintains the UOP fieldsfor each stream or channel based on the state of its reader module.Delay filter 112 updates the user operation bits whenever the state ofthe reader module changes.

Certain user operations might be permitted only in certain states of thedelay filter. Use of a prohibited operation might lead to inconsistentstates in the delay filter and user interface. For example, assume thata reader module that is performing “time-shifting” of multimedia contentis paused at point X in the circular buffer 124. In the meantime, writermodule 122 is still writing data into circular buffer 124. Depending onthe size of the circular buffer 124 and the time for which the readermodule has been paused, there is a possibility that writer module 122can catch up to the point X where the reader module is paused and try tooverwrite the data. In this case, delay filter 112 can forcibly“unpause” the reader module and send a notification to the applicationthat the paused reader module has been unpaused. There is a small timewindow between the time at which the reader is “unpaused” and the timeat which the application receives the notification. During this timewindow, the application might issue a prohibited user command, such as aforward scan. The reader module will now be performing the forward scanat the time the application receives the notification that the readermodule has been unpaused and is playing at normal speed. The applicationand the delay filter 112 are therefore in inconsistent states.

To avoid this problem, delay filter 112 allows the application to checkthe user operation bits whenever a user operation is issued to check forvalidity of the operation. Also, whenever the user operation bitschange, delay filter 112 reports this change to the application. Theapplication can therefore update its user interface to ensure thatinvalid user operations are not issued to the delay filter 112.

FIG. 15 is a flow chart of a software module or object that can becalled by the application through an application program interface(“API”), for example, whenever the user requests an operation or theuser operation status bits change. At step 430, the application waitsfor the user to request an operation. Once a user operation has beenrequested, the application checks the status of the corresponding useroperation bit, at step 431. If the operation is permitted at step 432,the application issues the requested operation to delay filter 112, atstep 433. If the requested operation is prohibited, the operation isrejected at step 434.

FIG. 16 is a flow chart illustrating steps performed by delay filter 112in maintaining the user operation bits for each status register. At step440, delay filter detects a change in the state of one of the readermodules. At step 441, the user operation bits of the status register forthe corresponding stream or channel are updated as a function of thechange in state. Delay filter 112 then notifies the application throughan API, for example, for that stream or channel of the change in state,at step 442. The application can then update the user interface todisable or enable selected subsets of the user operations listed in FIG.14. A user interface update can include setting or resetting lightindicators, issuance of an audible alert or changing screen displays,for example. The application can determine whether the user interfaceshould be updated by querying the delay filter 112 for the useroperation bits for a specific stream or channel.

Time Shifting

Another broad inventive aspect illustrated by the exemplary embodimentincludes the concept of “time shifting” when streaming information isrendered. Generally, an information appliance for receiving streaminginformation implements time shifting by including a buffer (in a oneembodiment circular buffer 124), writer module 122 which receives blocksof streaming information and writes blocks to the buffer, and at leastone reader module 126 which selectively reads the blocks from thebuffer.

Although all signal processing equipment inherently includes signalpropagation delay, “time shifting,” as used herein, is distinguishablefor the reasons discussed below separately or in any combination. In thefirst instance, the amount of time shifting (i.e., the relative positionof any one reader module 126 with respect to the writer module 122 incircular buffer 124) is selective and adjustable. In a first mode ofoperation, the user can “rewind”, “pause”, “fast forward” and “play” inany desired order, thereby changing the relative reading position of anassociated reader module 126 with respect to a position of writer module122 in circular buffer 124. Hence, rendered streaming information (whichbegins with a reader module 126) is delayed in time from the streaminginformation that is written by writer module 122.

In another mode of operation, logic is provided to adjust the amount of“time shifting” based upon operation of the system and without userintervention. For instance, if a reader module has been “paused” by auser, thus maintaining the current position of reader module 126 incircular buffer 124, at some time it may be necessary to begin advancingthe reader position in order that writer module 122 can store newstreaming information in circular buffer 124. This situation may requirereader module 126 to begin moving forward within circular buffer 124 ata rate greater than or at least equal to the rate at which streaminginformation is being written to circular buffer 124.

Automatic adjustment of the position of a reader module 126 in circularbuffer 124 can also be initiated by user commands not directlyassociated with movement of a position of reader module 122, such as“play”, “pause”, “rewind” and “fast forward”. As probably bestillustrated by example, the source of streaming information 114 to delayfilter 112 can be a multiple broadcast channel device such as atelevision tuner, or the like. If the user begins rendering informationpursuant to a selected broadcast channel, pauses, and then again resumesrendering information on the same broadcast channel, the associatedposition of reader module 126 will change position accordingly, therebyincreasing the relative distance of the position of reader module 126and the position of writer module 122 in circular buffer 124. If theuser then selects a different broadcast channel to be rendered, forexample, a different television channel, writer module 122 will thenrecord the streaming information associated with the new broadcastchannel in circular buffer 124. However, since the position of readermodule 126 is “time shifted” from the position of writer module 122 by adelay in proportion to the amount of time the user had paused, the userwill not notice the new broadcast channel selection until reader module126 reaches the new streaming information in circular buffer 124provided by the change in broadcast channel. In order to obviate theproblems presented by the foregoing, the position of reader module 126is automatically advanced to a position in circular buffer 124 adjacentthe position of writer module 122. This technique reduces the amount ofdelay so that rendered streaming information substantially followsbroadcast channel selection in a multiple broadcast channel environment.

Another distinguishing feature between “time shifting” as used hereinand typical signal propagation delay is the medium used to recordstreaming information. In particular, as discussed above, circularbuffer 124 can be embodied using any of the storage devices describedabove such as RAM memory, hard disk 27, or the like. In addition, theamount of memory present in circular buffer 124 is sufficient to providea perceptible delay to the user, if desired. In one embodiment, theextent of circular buffer 124 provides at least five minutes of delaybetween writing and rendering of streaming information. In a furtherembodiment, the extent of circular buffer 124 is sufficient to provideat least 30 minutes of delay between writing and rendering streaminginformation. In yet a further embodiment, the extent of circular buffer124 is sufficient to provide at least one hour of delay.

A further distinguishing feature of “time shifting” as used hereinincludes the presence of a single writer module 122 and a plurality ofindependent reader modules 126 ₁-126 _(N).

In one embodiment, any information received from streaming source 114 isfirst recorded in circular buffer 124 prior to rendering the streaminginformation to the user. In other words, all information to be renderedto the user is read with an associated reader module 126 from circularbuffer 124, wherein there exists no direct connection from streamingsource 114 to rendering devices 116. Although recording and reading fromcircular buffer 124 prior to rendering streaming information, mayslightly increase propagation delay when the position of reader module126 in circular buffer 124 is substantially adjacent to the position ofwriter module 122, advantages of always writing to and reading fromcircular buffer 124 include simplification in system design andoperation, which enhances system stability. Although a switchingmechanism could be provided in the system to switch from substantially“live” streaming information and time shifted streaming information asprovided by delay filter 112, the switching mechanism whether software,hardware or a combination thereof, nevertheless necessitates anothercomponent in this system that must be designed and tested to operatewith the remaining components of the system. Furthermore, by eliminatinga direct connection of the source of streaming information 114 and therendering devices 116 and always reading from circular buffer 124 priorto rendering, the quality of rendered information is consistent. Inother words, there is no change in quality (e.g., video quality or audioquality) of rendered streaming information since all information mustpass through the same system components. Furthermore, by always writingto and reading from circular buffer 124, the user need not remember toexecute a particular command so that streaming information is recorded,for example, in order to provide “instant replay.” In this manner, theuser is also able to retrieve and save an entire segment of streaminginformation (such as a television show) even when the user has alreadyrendered some of the streaming information. Thus, if the user is viewinga show and then later decides to save the show, he can invoke archivesystem 136 to begin at the start of the show in circular buffer 124 andtransfer a copy of the streaming information to a more permanent file.The latter can be performed while still viewing the show with a separatereader module 126. However, the user must initiate saving or archivingprior to writer module 122 writing over any of the desired streaminginformation, but in a preferred embodiment, circular buffer 124 is ofsufficient length to provide many minutes if not hours of storedinformation.

Data Block Structure

FIG. 17 is a pictorial representation of a block 500 of data stored incircular buffer 124 for one stream of a multi-stream source of streaminginformation. Generally, data block 500 includes a header portion 502, adata or payload portion 504 and a tail or end portion 506. Headerportion 502 contains relevant information as to the size of the datablock and the channel to which it pertains. In particular, headerportion 502 includes a field 508 in which the size of the entire block,including this field, is stored. Field 509 contains information used toidentify the streaming information to which the data block pertains.Field 510 provides a distance (typically bytes) from the beginning ofthe data block to where the actual streaming information or payload isstored in data block 500. Field 511 provides a value indicating the sizeof the actual data or payload 504. In one embodiment, each of fields508-511 are of fixed length in order that their location in data block500 is known.

In one embodiment, a field 512 is also provided in header portion 502.Field 512 provides other information relevant to the streaminginformation. As used herein, field 512 is referred to as “prefix field”in view of its relation to the actual data or payload 502. In a oneembodiment, prefix field 512 is of variable length, thus the informationneed not be length restricted. If desired, a field 513 is also provided,indicating the size of the prefix field 512. Typically, field 513 isalso of fixed length, like fields 508- 511.

Prefix field 512 can be used to store information such assynchronization indicators, discussed below, a pointer to the next datablock pertaining to the same stream, or other information as may bedesired. As appreciate by those skilled in the art, a format isspecified for information to be contained in prefix field 512 in orderthat relevant information can be decoded easily. However, since prefixfield 512 can be of any length, additional information can be stored asnecessary, thereby making data block 500 extensible.

It should also be noted that prefix field 512 can also include aselected amount of “fill” data. The fill data is used to adjust orincrease the size of data block 500 to a predetermined length, which maybe desirable for processing or storage in circular buffer 124.

Data or payload field 504 stores digital data received from the sourceof streaming information 114, which will be rendered.

In the embodiment illustrated, tail or end portion 506 includes twofields 520 and 521. Field 520, herein also referred to as “suffix field”is similar to prefix field 512 in that it can be used to storeinformation about data blocks that are successive or preceding in thesame stream. In one embodiment, suffix field 520 is at a known locationfrom the end of data block 500 and stores a pointer indicating thelocation of the preceding data block corresponding to the same channel.This information is particularly useful when it may be necessary toperform a “rewind” operation upon the streaming information contained incircular buffer 124. Suffix field 520 can also include other informationpresent in the data block 500 such as data associated with field 511 andfield 513. In a manner similar to prefix field 512, a format for theinformation contained in suffix field 520 is predetermined and known tothe reader module in order that relevant information can be obtainedtherefrom. Suffix field 520 can be of fixed length, or can be ofvariable length, wherein it may be desired to provide an additionalfield similar to field 513 that indicates the length of suffix field520. Suffix field 520 can also be used to store “fill data”, if desired.

Field 521 provides the size of data block 500 and, as such, is identicalto field 508 in header portion 502. Field 521 is useful when a readermodule progresses backward through the circular buffer 124. In thismanner, the reader module can ascertain the size of a data block 500 byreading the value contained in field 521, at which point, the readermodule has determined where the end of the preceding block is located.Without the presence of field 521, the reader module would otherwisehave to scan the data block 500 in order to determine its size orbeginning location, which can consume processing time.

FIG. 18 illustrates a sequence of data blocks 530 from a multi-streamstreaming information source. In particular, data blocks 532A, 532B and532A comprise a portion of a first stream; data blocks 534A and 534Bcomprise a portion of a second stream; and data blocks 536A and 536Bcomprise a portion of a third stream. The data blocks 532A-532C,534A-534B and 536A-536B together comprise a single channel and (e.g.,audio, video and data) are generally interposed such as illustrated inFIG. 18; however, there is generally no requirement that a specificorder be followed.

As indicated above, prefix fields 512 and suffix fields 520 can be usedto store pointers indicating preceding and succeeding data blocks ineach of the stream streams. In the illustration of FIG. 18, arrows 534represent that prefix fields 512 of data blocks 530 store pointers oraddresses of the immediate succeeding data block in each of the stream,respectively. Similarly, arrows 536 represent that suffix fields 320 ofeach of data blocks stores pointers or addresses of the immediatepreceding data block in each stream, respectively. In one embodiment,each of the prefix fields 512 stores the address of the prefix field 512in the immediate succeeding data block, while each of the suffix fields520 stores the address of the suffix field 520 in the immediatepreceding data block. In this manner, reader module 126 can quicklylocate and process data blocks for a particular stream of streaminginformation.

FIG. 19 illustrates a second sequence of data blocks 550. In sequence550, data blocks 552, 554 and 556 pertain to streaming informationreceived from streaming information source 114, which will beselectively rendered by rendering devices 116 in the manner discussedabove and further below. However, sequence 550 further includes datablocks 558 and 560 that “pad” the sequence 550 and do not contain anystreaming information to be rendered. Rather, data blocks 558 and 560are used, if desired, to organize or structure the sequence 550 storedin circular buffer 124, or other storage device, in a desired manner.For instance, it may be desirable that data blocks stored in circularbuffer 124 coincide with defined medium boundaries partition such assector or cluster boundaries. In FIG. 19, desired boundaries areindicated at 562A and 562B. Data blocks 552, 554 and 554 are organizedalong with a padding data block 560 of suitable size or length such thata data block 564 begins at the boundary 562B. If desired, a plurality ofpadding data blocks can be used, and the order of streaming informationdata blocks and padding data blocks can be changed to meet any desiredcriteria. The sequence of FIG. 19 is but one exemplary sequencestructure.

It should be noted that field 509 of each data block can be used toidentify padding data blocks from other streaming information datablocks wherein writer module 122 can generate padding data blocks asnecessary during processing and formation of streaming information datablocks. Furthermore, padding data blocks can be used in combination with“fill” data provided in prefix field 512 or suffix field 520, ifdesired. Mux formatter 138 and writer module 140 can also implement theuse of padding data blocks, and/or fill data contained in prefix field512 or suffix field 520.

Multiple Readers Per Buffer

It may be desirable, in some instances, to allow a plurality of users toaccess the streaming information at any one time. This can beaccomplished in any number of different ways. For example, the streaminginformation could be duplicated and one reader module can be providedfor each copy of the streaming information. However, this requires agreat deal of storage, and may also require increased processing powerfor making multiple copies of the streaming information and foraccessing the duplicate copies. Also, once the predetermined number ofcopies of the streaming information are in use, it can become verydifficult to add new users.

FIG. 20 is similar to FIG. 6 described above. However, FIG. 20illustrates portions of a system 610 in more detail and eliminates otherportions for clarity. For example, FIG. 20 illustrates that system 610includes circular buffer 124, a plurality of reader modules 614, 616,618 and 620 (which can be the same as or similar to reader modules 126mentioned above) associated with a plurality of users (users 630-636,respectively). Similarly, FIG. 20 illustrates that each reader moduleincludes an interface 622, 624, 626 and 628, respectively. Interfaces622-628 are described in greater detail below.

In the illustrative embodiment shown in FIG. 20, a plurality of users630, 632, 634 and 636 are depicted coupled to interfaces 622-628,respectively. In the embodiment illustrated, users 630 and 632 areviewers, or rendering applications, for viewing or rendering streaminginformation stored in circular buffer 124, while users 634 and 636 aredata storage sites which are used for archiving the streaminginformation stored in circular buffer 124. FIG. 20 also illustrates aseparate application program 638, separately coupled to interfaces622-628.

Again, it should be noted that FIG. 20 is illustrative only, and thatany number of users or application programs can be coupled to any numberof reader modules. Similarly, the users can be any suitable type ofusers desirous of accessing the streaming information stored in circularbuffer 124, and may be accessing the information for reasons other thanviewing, pr archival purposes, as is depicted in FIG. 20. Similarly, inaccordance with this aspect of the present invention, the bufferedstreaming information need not necessarily be stored in circular buffer124, but can be stored in a linear buffer, or any other type of buffer.However, circular buffer 124 is illustrated for purposes of simplicityonly.

In one illustrative embodiment, reader modules 614-620 are implementedas objects which conform to the COM object-oriented programming model.Each reader module or “reader object” has independent access to thebuffered streaming information. Each reader module 614-620 is depictedas accessing the streaming information at a different location oncircular buffer 124. This illustrates that reader modules 614-620 canaccess the buffered information at different times in the bufferedinformation stream. This allows the multiple users 630-636 toindependently employ all the features of time shifting mentioned above,without interfering with the other users accessing the bufferedinformation stream. One user, for instance, can archive the streaminginformation for later viewing, while at the same time another user canview the information.

Similarly, the archival user can store the buffered streaminginformation from a time or location in the data buffer which isdifferent than the time or location in the data buffer which the vieweris accessing. For instance, the archival user may simply be reading theinformation, as it is recorded in circular buffer 124, and storing it inarchival storage. By contrast, the viewer may be viewing the data, butintermittently pausing to take breaks or perform other tasks. Therefore,the reader module associated with the viewer may well be at a temporallydisplaced location in buffer 124 than the reader module associated withthe archival process.

Thus, FIG. 20 illustrates a system by which multiple reader modules canaccess the buffered streaming information. The system depicted requiresonly one writer module for buffering the streaming information, and onlya single copy of the streaming information. This greatly reduces theamount of storage required, and also reduces the processing powerrequired. Similarly, since the reader modules, in one illustrativeimplementation, are simply objects configured to access the informationin buffer 124, virtually any number of objects can be added to thesystem, limited only by the computational resources of the system. Thiscan be accomplished by simply instantiating another reader object.

Since the streaming information written in circular buffer 124 can beformed of one or more channels each having a plurality of streams (e.g.,audio, video, closed captioning, etc.), the output from each of thereader modules 614-620 will likely have a plurality of effective outputpins, each pin carrying one of the streams of a given channel in thestreaming information. Since system 610 illustrates that a plurality ofdifferent reader modules can be used to access the same bufferedstreaming information, this can present some obstacles which must beovercome.

For example, in some streaming architectures, the need may arise togroup the pins associated with each reader module 614-620 so that theuser, application, or whatever other component is receiving the datafrom the reader module, knows which pins are associated with that readermodule. An example of a streaming architecture which can be used inaccordance with the present invention is an architecture known asDirectShow services.

DirectShow services is an architecture which is commercially availableand well known. However, for a better understanding of certain aspectsof the present invention, a brief description of the DirectShow servicessystem may be beneficial.

DirectShow services refers to a modular system of pluggable componentsknown as filters, arranged in a configuration known as a filter graph. Acomponent referred to as a filter graph manager oversees the connectionof these filters and controls the flow of the streaming informationtherethrough. An exemplary filter graph is composed of a collection offilters of different types.

Most filters can be categorized into one of three types:

1. Source filters take data from a source, such as a disk file,camcorders, satellite feed, internet server, or VCR, and introduce thatdata into the filter graph;

2. Transform filters process data and pass it along to other portions ofthe filter graph, or out of the filter graph; and

3. Rendering filters render data to a hardware device or to any locationthat accepts media input (such as memory or a disk file).

In addition to these three types of filters, there are other kinds offilters also. Examples of other filters include effect filters, whichadd effects without changing the data type, and parser filters, whichare configured to understand the format of the source data and know howto read correct bytes, create time stamps, and perform seek operations.

Further, it is quite possible for some filters to represent acombination of filter types, or functions. In the DirectShowarchitecture, a filter is said to pass streaming information“downstream” to a next subsequent filter. An “upstream filter” refers tothe filter which passes data to the downstream filter, and a “downstreamfilter” refers to the next filter in line to receive the data. In oneillustrative embodiment, the filters are program modules written in anylanguage which can generate objects adhering to component object model(COM) programming. Of course, COM programming refers to theobject-oriented programming model which defines how objects interactwith a single application or between applications. In COM, clientsoftware accesses an object through a pointer to an interface (e.g. API)which has a related set of functions, called methods, on the object.

By way of example only, a filter graph 640, the purpose of which is toplay back MPEG-compressed video information from a file may take theform set out in FIG. 20A. Filter graph 640 includes source filter 642,MPEG parser 644, video decompression transform filter 646, audiodecompression transform filter 648, video render filter 650 and audiorender filter 652. Source filter 642 reads data from a disk and providesit as streaming information to MPEG parser 644. MPEG parser 644 parsesthe streaming information into its audio and video streams. Transformfilters 646 and 648 decompress the video and audio data in thecorresponding streams. Render filters 650 and 652 act to display thevideo data on a screen and send the audio information to a sound card,respectively.

It can thus be seen that, when using certain streaming architectures,such as the DirectShow architecture, it can be important thatapplications or other programming modules which are to receive streaminginformation from any of reader modules 614-620 know which output pinscorrespond to which of the reader modules. For example, since thestreaming information being read by each of the reader modules 614-620can include a channel having a plurality of streams, it is importantthat a user or application program receiving information from any givenreader receive all streams associated with that reader, and no otherreader. In other words, it would be undesirable for an applicationprogram (or a rendering filter, for example) to receive an audio streamfrom reader module 614 but a video stream and closed caption stream fromreader module 616. Reader module 616 may well be reading the streaminformation from a different location in buffer 124 than reader module614. Therefore, the audio and closed captioning streams would notcorrespond to the video stream.

In order to address this obstacle, one feature of the present inventionincludes a new COM interface which allows an object outside of readermodules 614-620 to enumerate the output pins which belong to thatparticular reader module. In other words, in the embodiment in whichreader modules 614-620 are implemented as objects, those objects areconfigured to expose methods through associated interfaces 622-628 tothe application (or other external components which have contact withthe reader modules) which allow manipulation of the object. Oneexemplary method exposed by interfaces 622-628 is referred to asIenumChannelStreamPin, which, when invoked by an outside object, yieldsa value which represents an enumeration of the particular output pinsbelonging to the particular reader module which was queried. In thisway, an outside object can quickly and easily obtain an enumeration ofthe particular pins belonging to any of the given reader modules614-620, through its associated interface 622-628.

Interfaces 622-628 also expose another method which allows any specificpin output by any of reader modules 614-620 to be queried for itsgroup's enumerator object. In other words, an outside object can queryany pin with which it has contact to obtain the identity of theparticular enumerator corresponding to the group which includes thatpin. In this way, a user or application program which has only recentlygained access to an output pin, can query that output pin to find theenumerator associated with that output pin. The external component canthen invoke the method exposed by the enumerator to obtain a completeenumeration of the pins corresponding to that channel. The externalcomponent can thus quickly and accurately obtain the identity of thepins associated with any given reader module, or channel, being outputby system 610.

Indexing and Seeking

As discussed above, indexer 132 generates an index which contains indexentries that are used for seeking. In other words, each of readermodules 614-620 can seek to any point in buffer 124 which has acorresponding entry in the index. In some instances, depending on thenature of the streaming information, substantially any point in buffer124 can be indexed. However, for other types of streaming information,it may be desirable to index only certain points within the streaminginformation. Such points are referred to herein as sync points. In otherwords, the nature of the streaming information may render itunreasonable or undesirable to seek to certain points within thestreaming information and begin rendering the data at that point.

For example, some video streaming information works on the well-knownMPEG2 video format. Such a format includes a number of different typesof frames which are referred to as I frames, B frames, and P frames. Iframes need no other information in order to be rendered, while B and Pframes are dependent frames which require information in a preceding Iframe in order to be rendered. In such an example, it may be undesirableto seek to a point within buffer 124 which begins with a B or P frame.In fact, many conventional video decoders may not even be able to decodedata which is provided from a B or P frame, without the necessaryinformation contained in the preceding I frame. Therefore, it may behighly desirable to index only I frames in such a video stream stored onbuffer 124.

Similarly, where the streaming information contains a stream, such aswritten text for closed captioning, it may be desirable to allow a userto seek to points which correspond to the beginning or ending of words,the beginning or ending of sentences, or the beginning or ending ofparagraphs. The same is true for an audio stream. In other words, it maybe desirable to allow a user to only seek to certain spots in the audiostream. Such spots may correspond to sentence or word boundaries, etc.

Therefore, one illustrative feature of the present invention includes astream analyzer 654, such as that illustrated in FIG. 21A. Streamanalyzer 654 is shown coupled to a source 656 of streaming informationand a sink 658 of streaming information. Stream analyzer 654 isconfigured to receive the streaming information from source 656 (whichcan be implemented as a software object).

In one illustrative. embodiment, stream analyzer 654 embodies knowledgeof the organization of the streaming information provided by source 656.Analyzer 654 also illustratively embodies knowledge regarding thedetection of logical boundaries in the information in order to obtainthe location of those boundaries (sync points). The location of the syncpoints can then be presented to any other software or hardware componentwhich may desire the information in order to enable skipping forward orbackward through the streaming information more practicable.

Stream analyzer 654 illustratively provides an indication of the syncpoints in one of two ways. In a first embodiment, stream analyzer 654embeds the location of the sync points within the streaming informationitself, as it is provided to information sink 658. This is referred toas in-band communication of derived sync point information.

In an alternative embodiment, stream analyzer 654 can provide the syncpoint information as indicated by dashed arrow 660, separately from thestreaming information provided to sink 658. This is referred to asout-of-band communication of the derived sync point information.

In one illustrative embodiment, stream analyzer 654 is implemented inthe C++ programming language using classes and communicating with othersoftware objects using COM interfaces. However, it should be noted thatsuch an implementation is illustrative only and the present invention isnot limited to this particular method of implementation.

The information sink 658 can, of course, be a downstream filter(downstream of stream analyzer 654) an application program, a renderingfilter or other program component, etc. Sink 658 can use the derivedsync point information itself, or simply pass it on to other componentswhich may wish to use the derived sync point information. In oneillustrative embodiment, the derived sync point information is providedto delay filter 112 which uses it in generating an index, as isdescribed in greater detail below.

In another illustrative embodiment, not only does stream analyzer 654analyze the incoming streaming information for logical boundaries whichcan be used as sync points, but stream analyzer 654 also analyzes theincoming streaming information for points of interest which may bedesirable sync points. Such points of interest can be substantially anypoints which are deemed to possibly be of interest to the user.Therefore, in such an embodiment, stream analyzer 654 embodies knowledgeof the type and format of information in the incoming stream, as well asknowledge of the types of events of interest to the user. Similarly,stream analyzer 654 is configured to contain knowledge of how to detectthese points of interest in the incoming streaming information, and isprovided with a mechanism to report the location in the data streamwhich corresponds to these points of interest.

FIGS. 21B and 21C are more detailed block diagrams illustrating streamanalyzer 654 in accordance with illustrative embodiments of the presentinvention. FIG. 21B illustrates an embodiment of stream analyzer 654 inwhich the derived sync point information is provided out-of-band, whileFIG. 21C illustrates an embodiment in which the derived sync pointinformation is provided in-band.

In FIG. 21B, stream analyzer 654 includes point of interest (POI)analysis component 662, point of interest interface component 664 andstreaming information interface component 666. The streaming informationis provided from source 656 to point of interest analysis component 662which calls on its base of knowledge regarding the type of informationand the organization of information contained in the streaminginformation, as well as its knowledge regarding points of interest tothe user. POI analysis component 662 then identifies points of interestin the streaming information and provides an output 668 which isindicative of the location of the points of interest (i.e., it isindicative of the sync points). Output 668 is provided to POI interface664 which makes the information available to sink 658. Similarly, POIanalysis component 662 passes the incoming streaming information on tostreaming information interface 666 such that the streaming informationis separately available to sink 658.

FIG. 21C is similar to FIG. 21B and similar items are correspondinglynumbered. However, rather than having two separate interfaces 664 and666 for the sync point information and for the streaming information,stream analyzer 654 illustrated in FIG. 21C has a single combinedinterface 669 which integrates the point of interest (or sync point)information into the streaming information and provides it as an in-bandoutput to sink 658. It should, of course, be noted that the integrationof the sync point information can also be accomplished in POI analysiscomponent 662. In any case, a single output of streaming information(which contains the sync point information in-band) is provided to sink658.

FIG. 21D is a more detailed block diagram of one embodiment of point ofinterest analysis component 662, in which in the integration of the syncpoint information into the streaming information provided at its output,is performed within point of interest analysis component 62. POIcomponent 662 includes, in the illustrative embodiment shown in FIG.21D, parsing component 670, written language analysis component 672,video analysis component 674 and audio analysis component 676. While anytype of information analysis components can be used, components 672, 674and 676 are illustrated for exemplary purposes only. POI analysiscomponent 662 also includes integration component 678.

Parsing component 670 receives the streaming information at its input.In the embodiment illustrated, the streaming information may include aplurality of streams (although only a single stream may be processed aswell), such as a written textual stream (e.g., closed captioninformation), a video stream and an audio stream. Parsing component 670parses the incoming streaming information into its respective streamcomponents and provides those components to appropriate analysis blocks672, 674 and 676.

Analysis components 672, 674 and 676 analyze the incoming streams,identify potential points of interest and generate informationindicative of the location of the those points of interest. Thestreaming information, as well as the sync point information, is thenprovided from each analysis component 672-676, to integration component678. In the embodiment illustrated, integration component 678re-integrates the streams into the original streaming information, andalso embeds the sync point information at appropriate locations withinthe streaming information. Thus, the in-band sync point information isprovided in the data stream at the output of integration component 678.

It should be noted that the analysis components 672-676 can look forsubstantially any desired points of interest. For example, writtenlanguage component 672 can be configured to look for sentence boundariesor word boundaries. In looking for sentence boundaries, the analysiscomponent can simply be configured to look for periods in the incomingstream. To look for word boundaries, component 672 can be configured tosimply look for spaces in the incoming stream. It should also be notedthat language analysis component 672 can be a more sophisticatedanalysis component, such as a natural language processing or naturallanguage analysis component, in which case component 672 can beconfigured to identify certain types of clauses, surnames, parts ofspeech, etc. In any case, information indicative of the written languagepoints of interest to be identified by component 672 is provided tocomponent 672 such that the desired points of interest can be identifiedin the incoming stream.

Similarly, video analysis component 674 can be used to identify anynumber of different points of interest. For example, component 674 canbe configured to identify commercial breaks, as well as to identify theI, B and P frames mentioned above. In identifying commercial breaks inthe incoming video stream, video analysis component 674 can simply beconfigured to look for a black screen which exists for a certainpredetermined amount of time. Similarly, in the MPEG2 video format, theI, B and P frames include markers identifying them as such, or arepreceded by a header identifying the frames. Therefore, video analysiscomponent 674 can be configured to look for the identifying informationin the stream.

Audio analysis component 676 can also be configured to look for a widerange of points of interest. By way of example, if the incoming streamis from a horror genre film, a war genre film, or an action genre filmor television program, audio analysis component can be configured tolook for loud noises, such as screams, gun shots, or car chase noises.In that instance, component 676 can simply be configured to look foramplitude information which exceeds a predetermined threshold.Similarly, component 676 can be a more advanced component, such as aspeech recognition or voice recognition component. In the instance inwhich component 676 includes a speech recognition component, it can beconfigured to look for sentence or word boundaries, or it can be used tolook for particular words or phrases. For example, component 676 can beused to identify words of profanity which can be marked for laterdeletion. Similarly, where component 676 includes a voice recognitioncomponent, it can be configured to identify the voice of certain actorsor actresses, or any other entity which can be so identified.

It should again be mentioned that the point of interest analysiscomponent 662 can be configured to identify the location ofsubstantially any point of interest which can be identified. Component662 simply needs to be provided with the information indicative ofpoints of interest to be analyzed, as well as information indicative ofthe type of information and format of information which will be receivedin the streams provided thereto. The specific embodiments mentionedabove are simply provided as examples, and are not to be viewed aslimiting the application of this feature of the present invention.

FIG. 22 is a flow diagram which illustrates the operation of POIanalysis component 662 in more detail. First, POI analysis component 662receives the streaming information. This is indicated by block 680.Parsing component 670 then parses the streaming information into itsrespective streams for a point of interest analysis. This is indicatedat block 682. At some point, prior to point of interest analysis, pointof interest analysis information indicative of the points of interest tobe identified must be provided to the various analysis components in POIanalysis component 662. This is indicated by block 684. The streams arethen analyzed based upon the received point of interest information asindicated by block 686.

Once the point of interest information is derived from the streams, thestreams are then reintegrated and the point of interest information isprovided either in-band or out-out-band, along with the integratedstreams. This is indicated by block 688. The data stream and the pointof interest information is then transmitted to a downstream component(such as delay filter 112) for further processing. Again, it should benoted that the point of interest information can either be providedin-band or out-out-band. This is indicated by block 690 in FIG. 22.

FIGS. 23A, 23B and 23C are more detailed flow diagrams illustrating theoperation of written language analysis component 672, video analysiscomponent 674 and audio analysis component 676. FIG. 23A illustratesthat written language analysis component 672 first receives the writtenlanguage stream information (such as closed caption information). Thisis indicated by block 692. Component 672 then analyzes the streaminformation to locate word or sentence boundaries, certain clauses,surnames or other parts of speech, etc. Again, the point of interestidentified can be substantially any points of interest for whichcomponent 672 is properly configured. This is indicated by block 694.Component 672 then provides an indication of the locations in thestreaming information where the points of interest reside. This isindicated by block 696.

FIG. 23B is similar to FIG. 23A, and similar items are correspondinglynumbered. However, rather than locating textual points of interest,video analysis component 674 analyzes the video stream information for ablack screen, image changes (which may correspond to desired scenechanges), I, B and P frame markers, etc. This is indicated by block 698.

FIG. 23C is similar to FIGS. 23A and 23B, and similar items aresimilarly numbered. However, rather than analyzing written language orvideo information, component 676 analyzes audio stream information. Inthe embodiment illustrated in FIG. 23C, component 676 analyzes the audiostream information for amplitude changes, sentence or word boundaries,certain content words (e.g., profanity), a certain persons voice, etc.This is indicated by block 700.

Index Generation

As described above, reader modules 614-620 in system 610 are configuredsuch that they can seek to any indexed point, in buffer 124. Forinformation streams which do not have sync points, it is reasonable forthe reader module to seek to substantially any sample within buffer 124.Therefore, in such information streams, any sample can be indexed. Insuch an embodiment, indexer 132 generates index entries which simplycorrespond to the desired granularity of the indexer. For instance, itmay be desirable in certain streams to index points which are temporallyspaced by approximately 0.25 seconds. With other information, it may bedesirable to index points which are spaced by two seconds or more. Ininformation streams which have no sync points, indexer 132 simplygenerates index entries identifying locations in buffer 124 which areseparated by the desired granularity of the indexer.

However, as discussed in the previous section, many data streams will beprovided with sync points. In those cases, it may be desirable forindexer 132 to generate index entries which correspond only to syncpoints. It will, of course, be noted that if sync points occur morefrequently than the desired granularity of the index generator, everysingle sync point may not be indexed. However, if sync points do notoccur more frequently than the desired granularity of the indexgenerator, substantially every sync point may be indexed.

Under such an arrangement, a problem can arise. For example, in a givenstreaming architecture, stream analyzer 654 described above may notexist, or it may not yet know whether the streaming information it isreceiving contains any sync points. That being the case, indexer 132 maynot know, at the time it begins receiving streaming information, whetherit must simply begin indexing samples according to its granularity, orwhether it is to wait to index only sync points. FIG. 24 is a flowdiagram illustrating the operation of indexer 132 in addressing thisproblem.

First, indexer 132 simply executes a query against stream analyzer 654to determine whether stream analyzer 654 can provide information as towhether the incoming streaming information contains sync points. This isindicated by block 702. Stream analyzer 654 may not even provide aresponse, or it may provide a response indicating that it does not yetknow whether the streaming information contains sync points. If analyzer654 provides a response indicating that information indicative ofwhether sync points exist is available, a flag is set. The flag isreferred to in FIG. 24 as the “Sync Point Info Is Authoritative” flag.This is indicated by blocks 703 and 704. If there is no informationavailable that is indicative of whether sync points exist, this flag isreset, as indicated by blocks 703 and 705.

Next, if information as to the presence or absence of sync points isavailable, it is determined whether any sync points are present. This isindicated by block 706. If stream analyzer 654 has already identifiedsync points in the incoming streaming information, it provides indexer132 with a response indicating that sync points do exist. In that case,indexer 132 sets a sync point flag to a value which indicates that theincoming streaming data does, in fact, contain sync points. This isindicated by block 707. However, if stream analyzer 654 does notrespond, or it has not yet located sync points in the incoming streaminginformation, and provides an output indicative of that to indexer 132,indexer 132 assumes, for the moment, that there are no sync points inthe incoming streaming information, and resets the sync point flag. Thisis indicated by block 708. Indexer 132 then receives a sample of thestreaming information as indicated by block 710.

Upon receipt of the sample, indexer 132 analyzes the sample to see ifthe sample has been marked as a sync point. This is indicated by blocks712 and 714. If the sample has not been marked as a sync point, indexer132 examines the sync point flag to see whether the sync point flag isset. This is indicated by block 716. If, at block 716, it is determinedthat the sync point flag is indeed set, that indicates that thestreaming information being processed does contain sync points, and thepresent sample is not marked as a sync point. Therefore, indexer 132does not index the sample under analysis but simply returns toprocessing at block 710.

If, however, at block 716, it is determined that the sync point flag isnot set, indexer 132 is still assuming that no sync points exist in theincoming streaming information. Therefore, the indexer simply determineswhether it should index the present sample under analysis, based on thedesired granularity of indexer 132. In other words, if indexer 132 is toindex points no more often than every one half second (for example),indexer 132 determines whether the present sample is temporally removedfrom the previously indexed sample by at least one half second. If so,the present sample is indexed. If not, processing simply returns toblock 710. This is indicated by block 718.

If, at block 714, indexer 132 determines that the present sample ismarked as a sync point, indexer 132 then determines whether the syncpoint flag is currently set. This is indicated at block 720. If the syncpoint flag is currently set, indexer 132 simply continues processing atblock 718 and determines whether it has reached sufficient temporaldisplacement from the previous index entry to index the present syncpoint.

However, if, at block 720 it is determined that, even though the presentsample is marked as a sync point, the sync point flag is not set, thenindexer 132 determines whether the “Sync Point Info Is Authoritative”flag is set, at block 721. If not, then indexer 132 realizes that it hasbeen assuming that no points exist in the streaming information underanalysis, but it also realizes that sync points do actually exist.Therefore, indexer 132 discards all the previous index entries (sincethey corresponded to non-sync point samples) as indicated at block 722and sets the sync point flag as indicated by block 724. Processing thenagain continues with respect to block 718. Similarly, if at block 721 itis determined that the “Sync Point Info Is Authoritative” flag is set,processing continues at block 718.

Therefore, it can be seen that by using the algorithm illustrated inFIG. 24, this feature of the present invention can be used to accuratelyindex points in the streaming information, regardless of whether theindexer currently knows whether the sync points are present in thestreaming information. Where sync points are not present, samples willbe indexed according to the desired granularity of the indexer. Whensync points are present, only sync points will be indexed.

Time Shift Seeking with Multiple Streams

As described above, the streaming information stored in buffer 124 caninclude multiple streams. As is also describe above, it may be desirableto allow reader modules 614-620 (illustrated in FIG. 20) to seek tovarious points in the streaming information indexed by indexer 132. Thispresents some difficulty. It is difficult to decide at which particularfile offset within buffer 124 the particular reader module should beginreading, after a seek is requested. This decision is made difficult by anumber of factors.

For example, the streams can be out of sync within the file located inbuffer 124. In other words, audio information can be stored before orafter video information, and in different size storage blocks, such thatthe two do not directly coincide. Also, the reader module may request toseek to a position in buffer 12 (to a sample with a particular timestamp) for which no sync point has been indexed. In other words, as setout above, indexer 132 does not index every single sample. Instead, itonly indexes based on its own granularity, or based on sync points.Therefore, a reader module may request to seek to a time stamp for whichno index entry is available. Further, some streams may contain syncpoints while others may not. In addition, not all sync points may beindexed, since they may occur more frequently than the desiredgranularity of the index. One illustrative feature of the presentinvention is directed to addressing these obstacles in seeking through achannel having multiple streams.

FIG. 25 is one exemplary illustration of a portion of buffer 124 whichis divided into a plurality of buffered samples. Samples 730, havingcross-hatching in a first direction, represent samples of a firststream. Samples 732, with cross-hatching in the opposite direction,correspond to samples of a second stream. Samples 734A, 734B, 734C and734D, with no cross-hatching, correspond to samples of a third stream.

In order to address the obstacles identified above, one illustrativeembodiment of the. present invention is a seek algorithm which isdescribed with respect to FIGS. 25 and 26. In the algorithm, theparticular reader module conducting a seek seeks to a highest fileoffset within buffer 124 which is suitable to satisfy each of thestream's individual seeking requirements. The reader module then beginsstreaming information, but drops information for each of the streamsbeing read which is read before the first sample that should actually beplayed for that stream. In addition, where no sync points with the exacttime stamp have been indexed, the nearest sync point before the desiredseek position is provided as a potential starting point, unless thenearest sync point before the desired seeking position is too far intothe past. Since some streams contain no sync points, the presentinvention assumes that, in a stream with no sync points, every sample isa sync point. Finally, since not all sync points may be indexed, theseeking algorithm first finds a nearest indexed sync point before andafter the desired seek position, and then determines the differencebetween the two indexed seek points to see whether there may be moresync points, which are not indexed, between the two. If so, the databetween the two indexed sync points is read to determine whetheradditional sync points (and ones closer to the desired seek point)exist.

More specifically, FIG. 26 is a flow diagram illustrating a seekingoperation in accordance with one illustrative feature of the presentinvention. In order to seek to a position within buffer 124 (illustratedin FIG. 25) the designated reader module first receives a seek requestto seek to a position in buffer 124 having a time stamp indicating timeT. This is indicated by block 740. In FIG. 25, the time stamp Tcorresponds to sample 734C. In the illustrative embodiment, sample 734Cis not indexed.

Next, a variable FILEOFFSET is set to infinity (or a very large number).This is indicated by block 742.

The first of the plurality of streams is then chosen for analysis. Thisis indicated by block 744.

For the first stream, a time variable T1 is set to the index entry withthe highest time stamp before the desired seek time T. In FIG. 25, itcan be seen that sample 734A for the designated stream corresponds tothe index entry T1 which has the highest time stamp before the desiredseek time T. If no such index entry exists, then T1 is set to T-Z. Inone illustrative embodiment, Z=100. This basically causes T1 to bediscarded as will be illustrated in later processing. This is indicatedby block 746.

Next, a time period T2 is set to an index entry for the selected streamwith the lowest time stamp after the desired seek time T. It can be seenthat sample 734D corresponds to a sample in the selected stream which isindexed, and which has the lowest time stamp to the right of time T(e.g., after time T). If no such index entry exists, the presentlyselected stream is ignored for purposes of computing the file offset,and a new stream is selected. This is indicated by block 748.

Thus, after processing at block 748, it can be seen that the two indexentries which lie on either side of time T, and which are closest totime T, have been identified. Those index entries correspond to times T1and T2. The temporal displacement between the desired time T and thepreviously indexed time T1 is represented by X=T−T1. The temporaldisplacement between the desired seek time T and the subsequent indexentry T2 is represented by Y=T2−T.

If both X and Y are greater than a predetermined time interval (e.g.,five seconds) that indicates that both of the indexed samples T1 and T2are too far from the desired seek position T to be helpful. Thisdetermination is indicated by block 750. If it is the case that X and Yare greater than the predetermined time interval, then the presentstream is simply ignored in computing FILEOFFSET. This is indicated byblock 752. Processing then continues at block 754 where it is determinedwhether additional streams exist. If so, a next stream is selected atblock 756 and processing continues with respect to block 746. If noadditional streams exist, a seek is performed at block 758, which isdescribed in greater detail below.

If, at block 750, it is determined that both values X and Y are notgreater than the predetermined time interval, then it is determinedwhether either X or Y are greater than the predetermined time interval.For example, at block 760, it is determined whether value X (whichcorresponds to T−T1) is greater than five seconds. If so, then the valueY must not be greater than five seconds, and a variableFILE_OFFSET_THIS_STREAM is set to index value T2. This is indicated byblock 762.

If, at block 760, it is determined that the value of X is not greaterthan five seconds, then it is determined whether the value Y is greaterthan five seconds. If so, then the value T2 can be ignored and thevariable FILE_OFFSET_THIS_STREAM is set to value T1. This is indicatedby blocks 764 and 766.

Basically, blocks 760-766 determine whether either index values T1 or T2are too far separated from the desired seek time T to be of help. Ifthey are, they are eliminated from further computing.

If, however, at block 764, it is determined that the value Y is notgreater than five seconds, that means that both samples corresponding tovalues T1 and T2 are close enough to desired seek time T to be ofinterest. That being the case, it is determined whether other syncpoints between T1 and T2 may exist. In other words, if a distancebetween T1 and T2 is so small that no other sync points could have beenindexed between them (based on the desired granularity of the indexer)then additional sync points may exist between samples T1 and T2 whichare closer to the desired seek position T, but which have simply beenomitted from the index because the granularity of the indexer was notsufficient to accommodate an additional index entry.

Therefore, a search algorithm is conducted in which the samples of theselected stream which reside between samples T and T1 are searched todetermine whether they are in fact sync points. If so, that would meanthat those sync points are closer in time to the desired seek positionT, and are prior in time to that seek position. If that is true, thenthe variable FILE_OFFSET_THIS_STREAM is set to the value correspondingto that identified sync point. This is indicated by blocks 768 and 770.

Therefore, at this point in the processing, the desired file offset forthe stream under analysis has either been chosen to be T1, T2, or avalue between T1 and T. Then, for the stream under analysis, the seekalgorithm requests subsequent filters in the streaming process to ignoreany data samples which are read out of buffer 12 and have time stampsprior to that identified by the variable FILE_OFFSET_THIS_STREAM. Thisis indicated by block 772.

Once the variable FILE_OFFSET_THIS_STREAM has been derived, it isdetermined whether that value is less than the current value of thevariable FILEOFFSET. If this is the first selected stream, the valuewill be less than the current value of FILEOFFSET, which has been set toinfinity. However, if this was a second, third, or later selectedstream, then the file offset for the present stream under analysis willbe used as the value for the variable FILEOFFSET only if it is furtherto the left (or earlier in time) than the current value for the variableFILEOFFSET. This has the effect of moving the seek point which willactually be used in seeking in buffer 124 to a point far enough left inthe buffer to satisfy every stream's individual seeking requirements.This is indicated by block 774.

Next, processing continues at block 754 where it is determined whetheradditional streams need to be analyzed. If so, processing continues atblock 756. If not, processing continues at block 758. In block 758, theselected reader module is commanded to seek to the sample identified bythe variable FILEOFFSET and start streaming data from buffer 124 at thatpoint. Since in block 772, a value has been set for each streamindicating that streamed data prior to the desired sync point for thatstream is to be ignored, the data streamed out of buffer 124 will not berendered for a given stream unless that data is coincident with, orafter, the earliest sync point identified for that given stream (andidentified by the variable FILE_OFFSET_THIS_STREAM).

One embodiment of pseudocode corresponding to the flow diagram of FIG.26 is set out below:  FileOffset = INFINITY  for (every connectedstream) {  T1 = Find the index entry with the highest timestamp beforeT;  if (T1 does not exist)   T1 = T − 100;//so that we ignore T1  T2 =Find the index entry with the lowest timestamp after T;  If (T2 does notexist)   Continue;//no index − ignore this stream in computing fileoffset  If ((T − T1 > 5sec) AND (T2 − T > 5sec))   Break;  else if (T −T1>5sec)//“too far”   FileOffsetForThisStream = FileOffset(T2);  Else if(T2 − T > 5sec)//“too far”   FileOffsetForThisStream = FileOffset(T1); Else {//neither one is “too far”   Search through the portion of thefile between FileOffset(T1) and FileOffset(T2)   To find the sync pointwith the highest time stamp that is still below T.   The position ofthat sync point becomes FileOffsetForThisStream.   Then tell thesplitter to discard any samples for this stream whose timestamps   aresmaller than the one we just found.   }   If (FileOffsetForThisStream <FileOffset)\    FileOffset = FileOffsetForThisStream;   }   Seek toFileOffset and start streaming.

Thus, it can be seen that the algorithm illustrated in FIG. 26 solves anumber of problems. With respect to streams that can be out of sync in afile, the algorithm seeks to the highest file offset suitable to satisfyevery stream's individual seeking requirements. The reader module thenbegins streaming at that point, but drops everything prior to the firstsample that should actually be played for each given stream. Similarly,where no sync point exists with the exact desired time stamp (thedesired seek position), the seek position is set to the nearest syncpoint before the desired seek position, unless the nearest sync point istoo far to the left (into the past). Also, since some streams maycontain seek points while others may not, every sample in a streamwithout seek points is deemed to be a suitable seek point. Finally,since not all seek points may be indexed, the seeking algorithm firstfinds the nearest indexed seek points before and after the desired seekposition and then looks at the difference between those indexed pointsto determine whether any sync points may reside between them. If so, thesamples to the left of the desired seek position are read to look foradditional sync points.

Although the present invention has been described with reference topreferred embodiments, workers skilled in the art will recognize thatchanges may be made in form and detail without departing from the spiritand scope of the invention.

1. A method of handling streaming information, comprising: receiving thestreaming information; analyzing the streaming information to locate oneor more points of interest in the streaming information; generating anindex of the one or more points of interest; delivering the index to auser separate from the streaming information; and controlling renderingof streaming information based on the points of interest.
 2. The methodof claim 1 and further comprising parsing the streaming information intostreams based on a type of data.
 3. The method of claim 1 wherein atleast one of the one or more points of interest in the streaminginformation relates to a frame in a video stream.
 4. The method of claim1 wherein at least one of the one or more points of streaminginformation relate to a particular word.
 5. The method of claim 1wherein at least one of the one or more points of interest in thestreaming information relates to a voice in an audio stream.
 6. Themethod of claim 1 wherein at least one of the one or more points ofinterest in the streaming information relates to a word boundary, asentence boundary or a paragraph boundary.
 7. The method of claim 1wherein the index is delivered as out-of-band communication.
 8. A methodfor handling streaming information, comprising: receiving streaminginformation related to a segment of multiple streams of streaminginformation into a buffer; rendering a portion of the segment; andarchiving the entire segment to a more permanent file after said portionof the segment has been rendered.
 9. The method of claim 8 and furthercomprising: determining a starting point of the segment.
 10. The methodof claim 8 and further comprising: receiving an indication from a userto archive the entire segment after said portion of the segment has beenrendered.
 11. The method of claim 8 wherein the segment is a televisionshow.
 12. The method of claim 8 wherein the segment includes at leasttwo of an audio stream, a video stream and a data stream.
 13. A computerreadable medium encoded with a data structure having a data block formatfor streaming information, the data block format comprising: a datablock having a header portion and an end portion; a first data blocksize field included in the header portion and indicating the size of thedata block; a second data block size field included in the end portionand indicating the size of the data block; and a payload field boundedby the first data block size field and the second data block size field,wherein the payload field includes streaming information to be rendered.14. The computer readable medium of claim 13 wherein the header portionincludes a stream identification field.
 15. The computer readable mediumof claim 13 wherein the header portion includes a field indicating alocation of the payload field in the data block.
 16. The computerreadable medium of claim 13 wherein the streaming information to berendered includes a plurality of streams.
 17. The computer readablemedium of claim 16 wherein the plurality of streams include at least twoof an audio stream, a video stream and a data stream.